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ABSTRACT 

The  Air  Force  Research  Laboratory,  Human  Effectiveness  Directorate  has  developed  a 
comprehensive  approach  to  mission  planning,  rehearsal,  and  training  called  Distributed  Mission 
Training  (DMT).  We  have  developed  a  high  fidelity  simulation  testbed  to  explore  the  impact  of 
principled  training  on  individual  and  team  performance.  The  information  obtained  in  our  testbed 
is  crucial  to  assessing  the  impact  of  the  Air  Force’s  ground  and  flying  training  and  rehearsal 
methods.  The  information  is  also  of  singular  importance  in  identifying  disconnects  and  shortfalls 
between  our  methods  and  the  operational  requirements  of  combat  pilots  and  teams.  This 
presentation  will  highlight  challenges  in  the  development  and  use  of  high  fidelity  simulation 
environments  for  enhancing  individual  and  team  performance.  We  will  focus  on  issues  related  to 
defining  mission  essential  competencies  for  aerospace  operations  and  on  the  use  of  these 
competencies  for  specifying  learning  objectives  and  for  developing  training  scenarios.  We  will 
also  highlight  recent  work  to  develop  valid  criteria  for  evaluating  the  performance  of  pilots  as 
individuals  and  as  part  of  a  four-ship  team  in  the  environment  and  the  relationship  of  this 
performance  to  actual  flying  operations. 


INTRODUCTION 

The  Air  Force  s  Distributed  Mission  Training  (DMT)  program  is  a  major  advance  in 
ground-based  training  that'  will  allow  pilots  and  other  warfighters  to  train  for  complex,  multi¬ 
player  combat  operations  using  a  network  of  flight  simulators  and  other  systems.  Chapman 
(1998)  states  that, 

The  DMT  network  will  be  created  incrementally,  focusing  first  on  team  training.  F-15C  and 
AWACS  Missions  Training  Centers  (MTCs)  will  be  the  first  DMT  constituents.  MTCs  will 
house  the  unit’s  simulators  or  mission  training  system,  briefing  and  debriefing  systems,  threat 
stations  and  simulations,  and  the  infrastructure  needed  for  local  and  wide  area  networking,  (p.  1) 

Researchers  from  the  Air  Force  Research  Laboratory,  Warfighter  Training  Research 
Division  (AFRL/HEA)  are  investigating  strategies  for  incorporating  DMT  into  advanced  flying 
training  in  operational  units.  When  reviewing  the  utility  of  an  earlier  generation  of  training 
systems,  Polzella,  Hubbard,  Brown,  and  McLean  (1987)  discovered  that  many  of  the 
instructional  features  that  had  been  incorporated  into  Air  Force  flight  training  simulators  were 
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unwanted  and  unused.  A  primary  goal  of  DMT  research  at  AFRL/HEA  is  to  provide 
commanders  and  instructors  at  MTCs  with  effective  and  validated  training  tools  and  materials 
including  training  protocols  to  meet  specific  needs,  sample  scenarios,  and  performance 
assessment  instruments.  The  laboratory’s  objective  is  to  assist  Air  Force  warfighters  use  DMT 
systems  to  effectively  and  efficiently  augment  flying  training. 

Background 

Research  on  training  effectiveness  of  multi-ship  simulation  systems  has  been  on-going  at 
AFRL/HEA  for  almost  ten  years.  In  addition  to  research  on  networked  simulation  technologies, 
activities  have  focused  on  application  of  DMT  for  continuation  training  of  fighter  pilots  and  air 
weapons  controllers.  Hoog  (1999)  described  how  DMT  systems  are  being  designed  to  enhance 
skills  that  warfighters  will  employ  both  as  individuals  and  as  members  of  a  team.  Effective 
application  of  multi-player  simulation  for  enhancing  individual  and  team  skills  has  been 
demonstrated  for  F-15  pilots  (Houck,  Thomas,  &  Bell,  1991),  F- 16  pilots  (Berger  &  Crane, 
1993),  Tornado  pilots  and  navigators  (Huddlestone,  Harris,  &  Tinsworth,  1999),  pilots,  forward 
air  controllers,  and  ground  forces  executing  close  air  support  (Bell,  et  al.,  1996),  and  Air  Force 
Special  Operations  teams  (Nullmeyer  &  Spiker,  in  press).  F-16  pilots  who  have  flown  in  a 
distributed  environment  have  rated  DMT  as  particularly  effective  for  training  four-ship  air-to-air 
employment  against  multiple  enemy  aircraft  (Crane,  Schiflett,  &  Oser,  2000).  F-16  pilots  have 
identified  individual  skills  including  radar  mechanization  (i.  e.,  using  the  various  modes  and 
capabilities  of  the  air-to-air  radar  to  detect,  track,  and  target  multiple  aircraft),  communication, 
and  building  situation  awareness  as  being  enhanced  by  DMT.  Team  skills  enhanced  by  DMT 
include  maintaining  mutual  support,  tactical  execution,  and  flight  leadership. 

One  of  the  most  critical  aspects  of  developing  high  performance  flying  missions  is  related 
to  the  identification  of  mission  essential  competencies  that  can  be  used  to  define  combat 
readiness.  The  Air  Force  has  attempted  to  specify  readiness  and  mission  performance 
requirements  at  a  number  of  levels  of  abstraction.  These  levels  of  abstraction  include  very 
specific  task  level  behaviors  that  must  be  performed  to  operate  fighter  aircraft.  They  also  include 
very  broad  statements  of  the  Air  Force  Mission  Essential  Tasks,  which  are  global  statements  of 
the  overall  missions  and  goals  of  the  Air  Force  and  its  component  organizations.  In  most  cases 
these  more  specific  task  level  behaviors  are  the  focus  of  Undergraduate  Pilot  Training  and  Initial 
Qualifications  Training  (IQT). 

From  the  time  an  Air  Force  pilot  completed  IQT  until  they  are  certified  as  “Combat 
Mission  Ready”  they  are  basically  required  to  completed  a  variety  of  formal  and  information 
training  and  operational  requirements  as  a  part  of  the  Ready  Aircrew  Program  (RAP).  However, 
the  specificity  of 

Crane  (1999)  reported  that  F-16  instructor  pilots  participating  in  DMT  research  would  frequently 
fly  as  flight  leader  for  the  morning  mission  but  would  then  assign  a  less  experienced  pilot  to 
serve  as  flight  lead  during  the  afternoon.  The  instructors  explained  that  these  pilots  were 
participating  in  Flight  Lead  Upgrade  (FLUG)  training  at  their  squadrons  and  that  DMT  was  a 
valuable  complement  to  aircraft  training.  As  applied  to  FLUG  training,  results  to  date 
demonstrate  that  experience  in  DMT  enhances  leadership  skills  for  multi-ship,  multi-bandit  air 
combat.  This  is  most  likely  a  function  of  both  DMT  system  capabilities  and  constraints  on 
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pilots  ability  to  train  in  the  aircraft  that  limit  opportunities  for  gaining  experience  over  a  wide 
range  of  tactical  situations.  Observations  of  pilots  during  DMT— FLUG  exercises  demonstrated 
that  increases  in  mission  performance  resulted  from  both  enhanced  skills  and  from  opportunities 
to  review  missions  and  to  assess  the  results  of  individual  actions. 

One  of  the  objectives  of  DMT-FLUG  research  was  to  identify  the  ways  in  which  DMT 
experience  could  increase  pilot  readiness  for  FLUG  missions.  These  examples  illustrate  two 
training  benefits  of  DMT.  The  first  is  providing  increased  opportunities  to  train  individual  skills 
that  must  take  place  in  a  multi-player  context.  Radar  sorting  and  targeting  requires  multiple 
friendly  and  enemy  aircraft;  opportunities  to  practice  this  skill  in  the  air  are  highly  limited.  DMT 
provides  many  opportunities  to  practice  this  skill  in  a  naturalistic  context  and  within  a  short 
period  of  time.  This  observation  is  in  agreement  with  Bell  (1999)  and  Schneider  (1989)  who 
propose  that  multiple  opportunities  to  practice  a  skill  in  a  variety  of  contexts  will  result  in 
substantial  performance  enhancement.  A  second  training  benefit  from  DMT  is  the  opportunity 
gain  experience  in  making  decisions.  Klein  (2000)  asserts  that  lack  of  experience  is  the  primary 
reason  for  poor  decision  making.  Without  high  levels  of  experience,  decision  makers  often  fail 
to  recognize  the  implications  of  various  situations  and  select  inappropriate  courses  of  action. 
Experience  in  DMT  can  augment  aircraft  experience  by  providing  pilots  with  multiple 
opportunities  to  execute  various  tactics  and  to  review  the  results  of  their  actions. 

In  the  next  iteration  of  the  DMT-FLUG  training  effort,  we  will  be  developing  more 
comprehensive  training  and  measurement  capabilities  that  will  permit  us  to  identify  which 
behaviors  improve  and  how  DMT  helped  in  the  improvement  process.  Our  findings  to  date 
indicate  that  minimum  capabilities  for  effective  DMT  include  the  following: 

•  DMT  systems  must  be  supported  by  programmable  scenario  generation  tools  that  permit  the 

incorporation  of  instructional  principles  and  training  strategies  to  foster  skill  development  and 
retention. 

•  Mission  replay  and  after-action  reviewing  capability  are  necessary  capabilities  for  an 
instructionally  viable  training  and  rehearsal  system. 

•  A  construct-oriented  individual,  team  and  mission  measurement  system  in  place. 


Simple  practice  or  free-play  are  not  efficient  training  strategies.  Programmable  scenario 
generation  tools  including  computer-generated  threats  with  the  capability  for  autonomous  action 
permit  instructors  to  take  advantage  of  findings  from  research  on  instructional  principles  to 
design  training  and  rehearsal  events  that  will  meet  specified  objectives.  In  multi-ship  air  combat, 
training  objectives  for  upgrading  pilots  include  learning  to  recognize  enemy  aircraft  formations 
and  selecting  an  appropriate  tactic,  communicating  the  plan  to  the  rest  of  the  flight,  executing  the 
plan,  and  changing  tactics  as  required.  The  programmable  scenario  generation  tools  in  DMT- 
FLUG  were  used  to  create  enemy  formations  and  tactics  that  varied  across  a  number  of 
dimensions  as  summarized  in  Table  2.  This  range  of  scenarios  combined  with  the  opportunity 
for  teams  to  fly  several  scenarios  in  an  hour  provide  upgrading  pilots  with  opportunities  to  gain 
experience  with  many  more  tactics  and  situations  than  would  be  possible  in  the  aircraft. 
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Another  key  element  in  the  success  of  DMT  is  the  replay  and  debrief  facility.  In  the 
AFRL  testbed,  each  cockpit’s  radar,  radar  warning  receiver,  head-up  display,  and  stores 
management  system,  are  recorded  and  played  back  synchronized  with  a  plan  view  display  and  all 
radio  communications.  The  flight  leader  can  pause,  rewind,  or  zoom  the  display  as  required. 
Using  the  replay  system,  pilots  can  review  the  information  that  was  available  to  them  inside  the 
cockpit  together  with  the  plan  view  display’s  depiction  of  the  complete  tactical  situation. 
Execution  errors,  poor  communication,  and  unplanned  contingencies  are  quickly  apparent. 
Debriefs  were  limited  to  90  minutes  for  scheduling  purposes  which  forced  teams  to  focus  on 
high  level  skills  such  as  communication  and  execution  without  getting  absorbed  into  analysis  of 
individual,  procedural  skills. 

Finally,  it  is  important  to  note  that  the  results  presented  in  this  report  represent  an 
evolution  of  our  training  paradigm  from  one  of  structured  practice  to  one  that  is  more  focused  on 
both  objectives-driven  instruction  and  construct  oriented  performance  measurement.  The  lessons 
learned  from  this  initial  DMT— FLUG  effort  have  provided  considerable  insight  into  the  problems 
and  needed  capabilities  associated  with  implementing  a  high  fidelity,  adaptive  training  and 
rehearsal  system  for  Air  Force  pilots.  The  following  summary  describes  future  directions  in 
research  being  undertaken  at  AFRL/HEA  based  on  the  results  of  DMT-FLUG. 

Training  needs  assessment  and  content  validation 

In  the  next  iteration  of  DMT-FLUG  research,  we  will  be  conducting  systematic 
assessment  of  training  needs  at  other  F-16  bases.  The  goal  of  the  needs  assessment  is  to 
elaborate  the  specific  tasks  and  mission  areas  that  are  problematic  for  pilots  in  upgrade  training. 
Additionally,  the  needs  assessment  will  provide  the  necessary  data  to  develop  an  instructionally 
principled  curriculum.  By  focusing  on  key  needs,  we  can  design  a  syllabus  that  fosters  the 
development  and  retention  of  skills  needed  for  FLUG  success.  Moreover,  the  needs  assessment 
will  complement  our  current  research  activities  in  the  definition  of  mission  essential 
competencies,  i.e.,  the  knowledge,  skills,  attitudes,  abilities  and  capabilities  required  for  combat 
mission  readiness.  This  effort  will  help  us  develop  a  more  comprehensive,  objectives-oriented 
syllabus  for  DMT-FLUG  and  identify  the  important  performance  parameters  associated  with 
mission  effectiveness.  It  will  also  help  us  in  the  development  of  metrics  to  quantify  training 
impact  in  terms  of  combat  mission  readiness  and  performance. 

Development  of  Performance  Criteria  and  Measurement  Instruments 

We  are  currently  in  the  process  of  developing  and  testing  evaluation  instruments  for 
individual  and  team  effectiveness  based  on  mission  essential  competencies.  By  adopting  a 
competency-based  approach  to  training  development  and  evaluation,  we  are  developing  data 
collection  and  analysis  methods  for  use  across  the  entire  spectrum  of  DMT  activities  (e.g., 
mission  preparation,  briefing,  execution,  and  debriefing).  Measurement  techniques  are  being 
developed  for  analysis  of  verbal  communications,  pilots’  mental  models,  team  processes,  and 
mission  outcomes. 

•  Content  analysis  methods.  Content-analysis  instruments  are  being  developed  to  evaluate 
verbal  communications  in  order  to  better  understand  the  quality  of  briefings,  within-mission 
communication,  and  debriefings.  In  addition,  measures  of  attitudes  toward  training. 
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motivation  and  perceived  application  and  utility  of  the  training  have  been  developed  for  use  in 
other  training  research  and  will  be  extended  to  FLUG. 

•  Mental  modeling  assessments.  We  are  also  conducting  a  series  of  studies  on  structured 
assessments  of  learning  such  as  situational  judgement  assessments,  mental  modeling 
Pathfinder,  concept  mapping,  and  multidimensional  scaling.  These  are  being  applied  to  assess 
learning  and  understanding  at  both  individual  and  team  levels. 

•  Process  and  outcome-oriented  measurement.  Using  mission  essential  competency 
definitions,  it  is  possible  to  identify  process-oriented  behaviors  that  are  associated  with 
effective  mission  performance.  We  will  then  be  able  to  develop  process-oriented  measures 
that  will  help  us  quantify  the  effects  of  DMT  experience  on  individual  and  team  knowledge 
acquisition,  crew  coordination/flight  integrity  (including  leadership  and  assertiveness) 
situational  awareness,  communication,  decision  making/risk  management,  task  management, 
and  mission  planning/debrief.  As  a  part  of  these  activities,  we  will  develop,  test,  and  validate 
protocols  to  capture  data  related  to  the  identified  processes.  The  key  process  behaviors  and 
metrics  will  be  used  to  further  refine  training  objectives  and  to  identify  opportunities  to 
improve  training  content  and  training  policy. 


Finally,  we  are  evaluating  the  reliability  and  validity  of  mission  outcome  measures  such 
as  “Turkey  Shoot”  and  “Top  Gun”  scoring.  Based  on  this  evaluation,  we  will  be  refining, 
automating,  and  applying  measures  for  simulator  and  flying  performance.  It  is  critical  that  any 
developed  measurement  system  provide  reliable  and  valid  data  to  demonstrate  that  training  has 
had  an  impact  on  actual  aircrew  competencies,  attitudes  toward  training,  motivation  to  train, 
expectations  regarding  training  and  transfer,  knowledge  mastery  and  subsequent  job 
performance.  Furthermore,  the  data  and  instrumentation  will  serve  both  a  formative  assessment 
function  by  providing  feedback  on  problem  areas  in  the  training,  and  a  summative  function  by 
permitting  cost  and  mission  impact  assessments. 

Summary 

DMT  research  has  demonstrated  both  the  utility  and  effectiveness  of  DMT  to 
complement  aircraft  training.  DMT  supports  operational  training  needs  training  by  providing 
opportunities  for  pilots  to  plan,  brief,  fly,  and  debrief  many  more  air-to-air  missions  than 
currently  possible  in  aircraft.  In  addition,  DMT  provides  instructors  the  capability  to  choose 
scenarios  that  exercise  selected  skills.  New  research  will  focus  on  developing  a  comprehensive 
training  syllabus  that  will  incorporate  needs  analysis,  specification  of  essential  competencies, 
scenario  development  and  selection  based  on  principles  of  instruction,  and  performance 
measurement  systems  that  will  assess  both  mission  outcome  and  team  processes. 
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Dr.  Brice  Stone 

The  U.S.  Air  Force  Space  Command  encompasses  over  30  career  fields  of  varying 
skill/knowledge  and  training  requirements,  which  provide  support  directly  or  indirectly  to  the 
mission  of  Space  Command.  In  today's  environment  of  tight  budgets  and  rapidly  changing 
technologies,  the  career  field  structure  and  training  requirements  within  the  auspices  of  Space 
Command  are  encountering  new  and  challenging  requirements.  Space  Command  is  directly  for 
the  training  and  career  field  management  of  two  AFSs:  space  systems  operations  (lC6xl  - 
enlisted)  and  space  and  missile  operations  (13Sx  -  officer).  These  new  requirements  for  the  two 
career  fields  under  Space  Command,  as  well  as  the  numerous  enlisted  career  fields  which 
support  Space  Command,  must  be  achieved  with  cost  effective  and  efficient  education  and 
training  programs.  Modification  and/or  enhancement  to  the  career  field  education  and  training 
program  must  provide  the  necessary  skills  for  accomplishing  the  job  and  tasks  of  the  present  and 
must  be  adaptable  to  the  needs  of  the  future.  Technologies  or  models,  which  allow  career  field 
planners  and  managers  to  carefully  evaluate  the  costs,  monetary  and  non-monetary,  of  changes  in 
training  needs,  are  essential  to  management  decision-making.  Such  technologies  as  the  Training 
Impact  Decision  Support  System  (TIDES),  the  Training  Effectiveness  and  Efficiency  Model 
(TEEM),  and  Job  Structuring  Technology  (JST)  provide  career  field  managers  with  tools  to 
evaluate  the  affects  of  career  field  structure  and  training  changes  from  the  perspective  of  costs, 
mission  readiness  and  productivity.  These  tools  form  the  basis  for  supporting  and  evaluating 
career  field  policy  modifications  and  changes,  as  well  as  providing  a  catapult  into  future 
evaluation  of  training  re-engineering  such  as  CBT  or  distance  learning. 

Training  Impact  Decision  Support  System  (TIDES) 

The  TIDES  system  allows  manpower  and  functional  managers  to  examine  trade-offs  between 
different  training  scenarios  for  enlisted  Air  Force  specialties  (AFSs).  TIDES  technology  helps  Air 
Force  training  managers  balance  an  occupation's  training  needs  versus  resources  and 
requirements.  This  balance  of  needs  and  requirements  allows  training  managers  to  optimize 
training  management  throughout  the  entire  career  of  an  individual  or  group  of  individuals  (this 
includes  all  training,  formal  and  informal,  as  well  as  all  jobs  an  individual  could  perform 
throughout  their  career  with  the  Air  Force).  Such  specialized  groups  of  individuals  have  different 
training  needs,  and  substantial  economics  are  possible  through  tailored  job  training.  TIDES 
technology  provides  analysts  and  decision-makers  with  a  tool  to  systematically  analyze  and 
integrate  information  about  jobs,  tasks,  career  assignments,  personnel  flows,  and  technical 
training  programs  within  an  Air  Force  occupation  (or  groups  of  related  occupations). 

By  dynamically  modeling  an  occupation's  career  flow  patterns,  the  TIDES  technology  provides  a 
"what  if  capability  to  assess  the  long-term  impact  of  current  and  future  constraints  stemming 
from  changing  training,  personnel  and  fiscal  policies  and  resources.  TIDES  analyses  also  aid 
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decision-makers  in  determining  what  tasks  associated  with  an  occupation  to  train,  when  to 
provide  that  training  (at  what  career  points),  and  the  method  best  suited  for  that  type  of  training. 

TIDES  “What  If  Modeling 


In  general,  TIDES  "what  if'  modeling  involves  creating  alternative  job  and  training  patterns 
reflecting  manpower,  personnel,  and  training  policy  options  of  interest.  This  can  be  done  by  a 
TIDES  analyst  or  modeler,  or  by  an  Air  Force  decision-maker,  such  as  the  functional  manager 
for  the  career  field  at  Headquarters  Air  Force  or  Major  Command,  or  in  a  group  session  such  as  a 
U&T  (Utilization  and  Training)  Workshop.  Once  an  alternative  is  specified,  the  model  is  run  on 
the  alternative  job  and  training  patterns,  and  model  results  are  compared  to  those  for  the  baseline 
model  (the  current  job  and  training  pattern). 

While  such  relationships  may  have  been  intuitively  obvious  to  some  managers  in  the  past,  it  is 
only  now  that  such  relationships  can  be  quantified  and  reported  to  support  and  justify  necessary 
training  investment  decisions.  The  types  of  information  and  data  displays  provided  in  a  decision 
support  system  such  as  the  TIDES  can  be  crucial  for  helping  managers  and  other  decision 
makers  improve  their  understanding  of  the  potential  impact  or  consequences  of  their  decisions. 
Further,  such  a  system  will  also  help  them  document  the  rationale  for  such  decisions,  and  defend 
such  decisions  to  higher  levels  of  command  within  the  Department  of  Defense  and  to  Congress. 

TIDES  Success 

During  the  development  of  TIDES,  tests  were  run  on  numerous  diverse  Air  Force  occupations: 
fairly  homogeneous  career  fields  such  as  the  B-l  Avionics  Test  Station  with  two  jobs  plus 
supervisors,  very  structured  specialties  such  as  Security  and  Law  Enforcement  with  85  job 
variations,  and  complex  electronics  career  fields  such  as  the  Radar  and  Inertial  Navigation 
Systems  technicians  with  many  distinct  jobs  by  equipment  system  each  with  its  own  advanced 
skills  training  course.  Occupations  with  high  volume  of  trainees  and  complex  tasks  were  also 
analyzed. 

One  of  the  occupations  studied  dealt  with  sophisticated  computers  and  switching  equipment 
systems  maintenance.  The  equipment  was  so  complex  that  the  training  and  personnel  were 
specialized  by  the  type  of  equipment  they  maintained.  Problems  were  beginning  to  show  up  at  a 
senior  level;  technicians  were  so  specialized  that  they  knew  little  about  the  entire  occupation. 
Therefore,  when  decisions  were  being  made,  certain  specialty  areas  may  have  been  under¬ 
emphasized  due  to  the  lack  of  the  decision-makers'  experience  with  some  equipment. 

TIDES  was  used  to  examine  various  ways  to  collapse  the  seven  separate  tracks  into  a  smaller  set 
of  subspecialties;  the  formal  consensus  among  representatives  of  the  various  constituencies  (at  a 
U&T  Workshop)  was  for  a  common  (general)  track  plus  three  specialized  tracks.  This 
consolidation  reduced  the  overhead  needed  to  manage  the  occupation  and  will  increase  the 
experience  base  for  future  senior  technicians.  Most  importantly,  when  combined  with  other  Air 
Force  personnel  policy  changes,  including  increasing  average  assignment  times  from  32  to  48 
months,  there  was  a  potential  reduction  or  avoidance  of  $1.7  million  in  annual  training  costs, 
even  though  each  new  individual  would  receive  more  complete  training  than  in  the  past. 
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A  study  was  conducted  within  the  Air  Force  security  police  career  field  investigating  the 
possibility  of  out-sourcing/privatization  parts  of  their  initial  skills  training.  A  proposal  was  made 
to  accept  applicants  with  associate  degrees  or  a  certification  program  in  place  of  conducting  full 
scale  ABR,  3  level  training.  The  potential  cost  savings  for  bypassing  the  course  was  determined 
for  management  review  and  consideration. 

Another  study  was  performed  for  OASD  to  determine  the  cost  savings  and  break-even  point  for 
initial  skills  and  7-level  training  for  three  diverse  AFSs  (aerospace  propulsion,  medical 
technicians,  and  pavements).  The  study  indicated  that  the  cost  savings  from  technology  insertion 
(primarily  CBT)  varied  significantly  due  to  several  key  factors  such  as  course  flow,  length  of 
course,  technical  rigor  of  subject  matter,  student  to  teacher  ratio,  etc.  The  results  indicated  that 
across  the  board  technology  insertion  was  not  economically  viable  requiring  a  course  by  course 
consideration  to  determine  the  economic  viability  of  re-engineering. 


TIDES  Current  Efforts 

The  TIDES  technology  runs  in  a  PC-based,  MS  Windows  environment.  Once  an  initial  TIDES 
data  base  for  a  specialty  has  been  developed,  TIDES  users  are  able  to  sit  at  their  personal 
computers,  formulate  and  run  various  scenarios  weighing  the  alternatives,  and  forecast  the 
impacts  of  decisions  before  they  become  policy.  The  advanced  work  includes  a  template 
(software  [MS  Word]  )for  Career  Field  Education  &  Training  Plans  (CFETPs)  to  service  a  HQ 
USAF/DPPE  need.  The  payoff  to  the  USAF  is  a  systematic  method  to  enable  career  field, 
functional,  and  training  managers  to  maximize  efficiency  and  training  effectiveness  while 
minimizing  training  costs,  yet  still  provide  the  means  to  produce  the  highest  quality  mission- 
ready  trained  forces. 

Training  Effectiveness  and  Efficiency  Model  (TEEM) 

TEEM  was  developed  for  the  Human  Resources  Directorate  of  the  Armstrong  Laboratory  as  a 
methodology  that  facilitates  the  use  of  training  evaluation  data.  TEEM  links  information  from 
the  field  to  information  from  the  course  in  an  automated  process  and  displays  that  information  in 
a  graphical  manner,  which  then  allows  course  designers,  developers,  and  decision-makers  to 
view  and  identify  data  trends.  Specifically,  it  is  a  computerized,  task-based  training  evaluation 
methodology  that  integrates  Occupational  Survey,  course,  efficiency,  effectiveness,  and  transfer 
data  at  the  same  level  of  specificity  to  facilitate  course  revisions  and  improve  training  quality. 
TEEM  allows  users  to  manipulate,  display,  and  print  all  results  produced  by  this  methodology. 

Training  efficiency  examines  whether  the  appropriate  amount  of  time  is  devoted  to  training 
tasks,  or  whether  tasks  are  potentially  over-  or  under-  trained.  Task-level  occupational  survey 
information,  such  as  Task  Learning  Difficulty,  which  is  based  on  the  notion  of  time  to  learn,  is 
compared  to  the  actual  amount  of  time  devoted  to  training  each  task  in  a  course.  The  results  are 
then  graphically  depicted  as  one  of  three  outcomes.  A  training  match  is  depicted  if  the  estimated 
time  to  learn,  for  example,  is  relatively  the  same  as  the  actual  time  devoted  to  training  that  task, 
indicating  that  the  relative  amount  of  time  devoted  to  training  that  task  seems  appropriate.  A 
potentially  over-trained  task  is  depicted  if  relatively  more  time  is  spent  training  a  task,  compared 
to  the  corresponding  time  to  learn  estimate,  indicating  that  training  time  might  be  reduced  for 
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those  tasks.  A  potentially  under-trained  task  is  depicted  if  the  time  to  learn  estimate  is  relatively 
greater  than  the  corresponding  actual  training  time,  indicating  that  more  training  time  needs  to  be 

Sttir  THUS’  ^  ^  "  tr3ining  "  disP^d  «  one  of 


i n formation  t  -  matchmS  outcomes  are  then  integrated  with  training  effectiveness 
information.  Training  effectiveness  examines  whether  course  graduates  perform  tasks  above  or 

below  a  specified  standard  later  on  the  job.  A  2  X  3  matrix  is  then  utilized  to  display  the 
ctiveness  and  efficiency  integration.  Two  columns  represent  the  training  effectiveness  for 
each  trained  task  as  above  or  below  the  training  standard.  Three  rows  represent  the  three 

notent!!  l°UtCrtfr0nJ  th^ffLc‘enc>'  results:  a  matctF  a  potentially  over-trained  task,  and  a 
potentially  under-trained  task.  Thus,  a  task  may  fall  into  any  of  the  six  boxes  of  this  2X3 

matrix^  For  example,  a  task  performed  below  standard  may  also  be  under-trained,  indicating  that 
more  training  time  might  be  allocated  to  that  task  to  improve  its  effectiveness. 


Finally,  efficiency  and  effectiveness  results  are  integrated  with  training  transfer  data.  Training 
transfer  examines  whether  or  not  learning  transfers  from  the  course  to  the  job.  Efficiency  is 
measured  in  terms  of  the  average  number  of  times  each  task  is  performed  on  the  job 
Effectiveness  is  measured  in  terms  of  graduate  abilities  to  apply  what  was  taught.  This 
information  is  integrated  into  the  2  X  3  matrix  and  is  displayed  with  the  tasks  that  fall  in  each 
box  to  allow  course  designers  to  see,  in  one  place,  all  of  the  data  that  are  related  to  each  task. 

Examined  separately,  the  results  of  training  efficiency,  training  effectiveness  and  training 
transfer  data  provide  useful  information  for  understanding  and  changing  different  aspects  of  a 
training  system.  Results  of  the  matching  technique  suggest  that  the  course  efficiency  can  be 
improved  by  reducing  training  time  for  over-trained  tasks  and  increasing  training  time  for  under¬ 
trained  tasks.  Training  effectiveness  data  suggest  that  for  tasks  performed  below  the 
performance  standard,  course  revisions  may  be  necessary  while  tasks  performed  above  standard 
are  less  likely  to  require  course  changes.  Training  transfer  data  indicate  that  more  task 
experience  might  be  required  early  on-the  job  to  facilitate  job  performance  where  the  opportunity 
to  perform  the  trained  tasks  is  low.  However,  these  date  are  more  useful  when  considered  in 
combination.  For  example,  the  results  illustrate  how  a  training  program  can  be  re-designed  to  be 
more  efficient  by  reducing  training  time  for  over-trained  tasks  that  were  performed  above 
standard,  while  increasing  training  time  for  under-trained  tasks  that  were  performed  below 
standard.  Furthermore,  the  training  transfer  information  is  useful  in  determining  whether  the 
training  program  itself  should  be  modified,  or  the  transfer  environment  changed.  For  example,  if 
a  task  is  under-trained,  the  amount  of  practice  on  the  job  is  sufficient,  but  it  is  performed  below 
standard,  the  training  program  might  be  modified  by  increasing  the  training  time  for  that  task 
Alternatively,  if  an  over-trained  task  is  performed  below  standard  on  the  job,  yet  the  amount  of 
practice  for  that  task  is  low,  modifying  the  training  program  is  probably  not  a  reasonable  action. 
A  better  solution  might  be  to  ensure  training  graduates  receive  more  practice  for  that  task  on  the 
job,  or  consider  eliminating  it  from  training,  and  allocate  that  time  to  other  tasks  in  the  program. 

Of  course,  other  options  should  always  be  considered,  such  as  developing  improved  instructional 
approaches. 
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Job  Structuring  Technology  (JST) 

The  Job  Structuring  Technology  (JST)  currently  being  developed  for  AL/HRMJ  examines 
possible  AFS  mergers  or  restructuring  based  on  task  modules.  Such  modules  are  semantically 
linked  to  knowledge  domain  lexicons  developed  from  Office  of  Personnel  Management  (OPM) 
civilian  series  descriptions.  This  technology  has  the  goal  of  targeting  specific  occupational  areas 
and  evaluating  potential  scenarios  to  optimize  force  structuring  based  on  Mission-Essential 
Modules  (MEMs)  of  tasks  (Driskill,  Weissmuller,  Moon  &  Black,  1995).  Support  tools 
developed  for  use  in  this  project  include  a  Historical  AFSC  Information  Locator  System 
(HAILS)  to  track  specialty  changes  across  time,  lists  of  current  occupational  analysis  and 
research  studies  and  requirements,  and  eventually  will  include  CFETP  and  other  required 
documents.  The  prototype  HAILS  has  already  been  delivered  to  AL\HRMJ  as  well  as  to  the  Air 
Force  Occupational  Measurement  Squadron  for  operational  use  as  an  AFS  reference  system. 


This  project  developed  the  prototype  of  the  Structuring  Technology  (JST),  a  microcomputer- 
based  system  that  simulates  the  processes  and  activities  of  force  planners;  mission  designers  and 
planners,  manpower,  classification,  assignment-placement,  and  promotion  personnel;  and 
training  planners  when  changes  in  officer  and  enlisted  specialty  classification  structures  appear 
to  be  warranted.  Underlying  the  system  is  the  premise  that  the  primary  requisite  of  classification 
structure  should  be  to  support  Air  Force  mission  requirements.  In  the  broadest  sense, 
classification  structure  either  drives  or  heavily  influences  force  planning,  such  personnel 
functions  as  assignment  and  career  progression,  manpower  planning,  and  training  design  and 
delivery.  The  JST  is  a  decision  aid  to  support  design  or  restructuring  of  classification  structures 
that  provide  the  most  efficient  and  effective  support  of  mission  requirements  through  the 
evaluation  of  impacts  of  the  structure  on  the  force  management,  manpower,  personnel,  and 
training  functional  requirements. 

The  system  employs  an  array  of  data,  available  from  a  variety  of  Air  Force  databases,  and 
incorporates  software  to  apply  these  data  to  prototype  as  well  as  evaluate  and  trade-off  among 
classification  structure  options  from  which  the  various  personnel  must  make  operational 
decisions.  Design  features  include  such  capabilities  as  assessing  changes  among  a  few  (two  or 
three)  Air  Force  Specialties,  large-scale  reorganization  (comparable  to  the  reorganization  in  the 
immediate  past),  and  creation  of  new  specialties. 

The  JST  software  includes  state-of-the  art  semantic-assisted  analytic  techniques  for  satisfying  a 
basic  requirement  of  any  classification  methodology.  Specifically,  aside  from  the  requirement  to 
support  mission  requirements,  a  fundamental  reason  for  classifying  a  large  variety  of  work 
requirements,  especially  diverse  work  requirements,  into  some  structure  is  to  maximize 
productivity.  This  purpose  is  most  efficiently  satisfied  when  workers  not  only  are  trained  most 
cost  effectively,  but  once  trained  can  migrate  from  job  to  job  within  a  work  category  with  a 
minimum  of  on-the-job  training  and  loss  of  productivity.  Implicit  in  this  purpose  is  the  need  to 
"classify"  jobs  according  to  the  commonality  or  homogeneity  of  knowledge,  skill,  and  ability 
requirements,  not  only  at  general  levels  of  commonality  but  at  the  most  specific  level  possible. 
The  semantic-assisted  software  in  the  JST  is  designed  to  permit  definition  of  homogeneity  at 
different  levels,  including  the  specific  level  from  which  optional  scenarios  of  classification 
structure  can  be  generated  and  evaluated  for  impacts  on  the  other  functions  as  well  as  operational 
utility. 
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The  JST  analyzes  the  commonality  of  work  requirements  within  each  of  the  job  descriptions. 
The  system  provides  methods  for  reading  each  of  the  requirements,  assigning  weights  to  the 
objects  worked  on  and  the  actions  taken,  and  using  these  weights  to  assess  the  homogeneity 
and/or  diversity  of  work  through  comparison  of  how  similar  work  is  structured  by  well- 
recognized,  accepted,  and  time  tested  classification  methodologies.  The  system  also  can 
compare  among  different  job  descriptions.  It  is  during  this  process  that  the  JST  semantic- 
assisted  analytic  software  not  only  reads  job  requirements,  but  also  interprets  among  them  in 
terms  of  those  that  are  most  homogeneous. 

Further,  JST  processes  provide  planners  (including  the  classifier  as  well  as  other  functional 
managers)  the  opportunity  for  the  generation  of  alternative  ways  to  distribute  the  body  of  work 
required  by  a  mission  manager.  Each  of  the  options  then  can  be  assessed  with  regard  to  its 
impact  on  the  other  essential  planning  functions  (e.  g.,  manning,  training),  compared  so  that 
potential  tradeoffs  are  identified,  and  an  operational  decision  about  classification  of  jobs  can  be 
made. 


The  JST  is  appropriate  for  considering  such  decision  possibilities  as  mergers  of  two  or  more 
specialties,  separation  of  one  specialty  into  multiple  specialties,  or  distribution  of  the  work  of  a 
specialty  into  other  existing  or  newly  created  specialties.  In  each  case,  specialty  descriptions  can 
be  read,  weights  computed  (based  on  within-specialty  importance),  commonality  of  work 
requirements  assessed,  optional  configurations  generated,  impacts  of  the  options  evaluated, 
tradeoffs  identified,  and  revised  or  new  structure  decided  upon. 

Integrated  Training  Technologies  (INTTECH) 


The  military  and  civilian  training  communities  have  a  continuing  need  for  innovative  analysis 
and  modeling  tools,  which  improve  the  ability  to  capture  and  more  accurately  access  the 
responses  of  individuals  and  systems.  At  the  core  of  many  simulations  or  analyses  are  models, 
which  embody  changes  in  such  responses  to  current  or  expected  training  requirements  and 
conditions.  The  approach  to  building  these  models  or  representations  is  similar  whether  the 
problem  is  the  identification  of  training  emphasis,  the  maintenance  of  a  turbine  engine  from  its 
expected  condition,  the  reaction  of  military  personnel  to  changes  in  career  training  requirements, 
or  the  maintenance  and  control  of  complex  computer  systems  supporting  weapons  systems. 
Recent  advances  in  modeling  and  simulation  techniques,  occupational  data  collection,  and  model 
selection  methods  have  suggested  more  powerful  approaches  for  capturing  richer  and  more 
realistic  estimates  of  impacts  for  changes  in  training  programs  and  personnel  policies. 
INTTECH  addresses  the  need  for  a  common  tool  (platform)  to  provide  sophisticated  analysis, 
representation,  and  modeling  of  training  and  training  policy  and  recognizes  the  requirement  that 
such  a  tool  be  capable  of  inter-operation  with  existing  and  planned  training  policy  tools. 

Through  many  years  of  research,  Armstrong  Laboratory  and  its  collaborators  developed  and 
evaluated  methodologies  and  techniques  to  capture  the  effects  of  changes  in  training  programs, 
requirements,  and  policies.  Most  of  the  prototype  technologies  developed  have  been  to  support 
and  improve  of  Air  Force  specialty  (AFS)  decision  making  on  training  and  utilization  policies. 
Ultimate  responsibility  for  decisions  on  AFS  training  rests  with  the  Air  Staff  Career  Field 
Manager  (CFM)  but  such  decisions  are  usually  supported  by  the  advice  and  concurrence  of 
MAJCOM  functional  managers,  technical  training  managers,  occupational  analysts,  and  military 
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personnel  classification  personnel  through  Utilization  and  Training  Workshops  (U&TWs).  This 
integrated  decision  making  process  is  now  mandated  under  Air  Force  Manual  36-2245, 
Managing  Career  Field  Education  and  Training,  30  June  1995. 

TIDES,  TEEM  and  JST  described  above  are  based  on  various  levels  of  task  information,  some  at 
the  task  module  level  and  some  with  linkages  to  other  kinds  of  information  (required 
knowledges,  equipment  used,  skills  required,  etc.).  To  be  useful  to  decision  makers,  such  data 
must  be  easily  transportable  among  the  various  technologies,  which  requires  development  of 
software  capable  of  handling  multiple  levels  of  task  data  (tasks,  task  modules,  metamodules, 
etc.).  Additional  research  and  development  is  needed,  during  the  development  of  the  integrating 
software  platform,  to  determine  appropriate  statistics  needed  when  moving  from  task-specific  to 
more  generic  levels,  or  across  occupational  areas. 

Implementation  of  emerging  training  technologies  into  a  common  facility  involves  exploratory 
research  into  the  methodology  for  communicating  among  the  technologies  to  be  resident  on  the 
system,  prototyping  of  the  facilities,  and  final  development  and  implementation  of  the  facilities 
into  a  new  Integrated  Training  Technologies  (INTTECH)  platform.  From  a  research  perspective, 
there  are  a  number  of  challenges  that  must  be  addressed  to  link  the  training  R&D  technologies 
together  in  a  common  platform.  One  of  the  greatest  challenges  is  related  to  determining  the 
appropriate  level  or  unit  of  analysis  that  serves  as  the  basis  for  linking  the  initial  technologies 
together.  TEEM,  TIDES  and  JST  are  technologies  that  examine  career  field  training  at  distinctly 
different  levels  of  analysis.  The  TEEM  is  designed  to  provide  information  for  evaluating  formal 
training  courses  within  a  career  field.  TEEM  focuses  on  the  more  micro-level  aspects  of  the 
career  field  training.  TIDES  was  designed  to  examine  and  model  job  and  training  requirements 
for  an  entire  career  field.  That  is,  at  a  more  macro-level.  Output  from  TIDES  could  be  provided 
to  TEEM  as  course  requirement  specifications.  Output  from  TEEM  can  be  incorporated  in  the 
TIDES  for  formal  course  design  and  revision.  Additionally,  TEEM  output  might  provide  critical 
information  for  validating  training  time  and  training  allocation  models  in  TIDES. 

INTTECH  provides  a  basis  for  identifying  and  testing  alternative  linkage  approaches,  assessing 
the  impact  of  integration  on  the  quality  of  analysis  results  from  the  actual  technologies,  and 
developing  and  demonstrating  a  common  technology  platform.  INTTECH  will  systematically 
link  and  integrate  the  three  technologies  such  that  information  and  data  could  be  dynamically 
shared  among  manpower,  personnel,  job,  and  training  technologies  together  in  a  common 
platform  similar  to  that  which  is  available  in  a  software  suite  such  as  Microsoft  Office. 
INTTECH  will  provide  the  basis  for  supporting  and  evaluating  career  field  policy  modifications 
and  changes,  as  well  as  providing  a  catapult  into  future  evaluation  of  training  re-engineering 
such  as  CBT  or  distance  learning. 
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unwanted  and  unused.  A  primary  goal  of  DMT  research  at  AFRL/HEA  is  to  provide 
commanders  and  instructors  at  MTCs  with  effective  and  validated  training  tools  and  materials 
including  training  protocols  to  meet  specific  needs,  sample  scenarios,  and  performance 
assessment  instruments.  The  laboratory’s  objective  is  to  assist  Air  Force  warfighters  use  DMT 
systems  to  effectively  and  efficiently  augment  flying  training. 

Background 

/ReSCarCh  °n  training  effectiveness  of  multi-ship  simulation  systems  has  been  on-going  at 
AFRL/HEA  for  almost  ten  years.  In  addition  to  research  on  networked  simulation  technologies, 
activities  have  focused  on  application  of  DMT  for  continuation  training  of  fighter  pilots  and  air 
weapons  controllers.  Hoog  (1999)  described  how  DMT  systems  are  being  designed  to  enhance 
skills  that  warfighters  will  employ  both  as  individuals  and  as  members  of  a  team.  Effective 
application  of  multi-player  simulation  for  enhancing  individual  and  team  skills  has  been 
demonstrated  for  F-15  pilots  (Houck,  Thomas,  &  Bell,  1991),  F- 16  pilots  (Berger  &  Crane, 
1993),  Tornado  pilots  and  navigators  (Huddlestone,  Harris,  &  Tinsworth,  1999),  pilots,  fonvard 
air  controllers,  and  ground  forces  executing  close  air  support  (Bell,  et  al.,  1996),  and  Air  Force 
Special  Operations  teams  (Nullmeyer  &  Spiker,  in  press).  F-16  pilots  who  have  flown  in  a 
distributed  environment  have  rated  DMT  as  particularly  effective  for  training  four-ship  air-to-air 
employment  against  multiple  enemy  aircraft  (Crane,  Schiflett,  &  Oser,  2000).  F-16  pilots  have 
identified  individual  skills  including  radar  mechanization  (i.  e.,  using  the  various  modes  and 
capabilities  of  the  air-to-air  radar  to  detect,  track,  and  target  multiple  aircraft),  communication 
and  building  situation  awareness  as  being  enhanced  by  DMT.  Team  skills  enhanced  by  DMT 
include  maintaining  mutual  support,  tactical  execution,  and  flight  leadership. 

One  of  the  most  critical  aspects  of  developing  high  performance  flying  missions  is  related 
to  the  identification  of  mission  essential  competencies  that  can  be  used  to  define  combat 
readiness.  The  Air  Force  has  attempted  to  specify  readiness  and  mission  performance 
requirements  at  a  number  of  levels  of  abstraction.  These  levels  of  abstraction  include  very 
specific  task  level  behaviors  that  must  be  performed  to  operate  fighter  aircraft.  They  also  include 
very  broad  statements  of  the  Air  Force  Mission  Essential  Tasks,  which  are  global  statements  of 
the  overall  missions  and  goals  of  the  Air  Force  and  its  component  organizations.  In  most  cases 
these  more  specific  task  level  behaviors  are  the  focus  of  Undergraduate  Pilot  Training  and  Initial 
Qualifications  Training  (IQT). 


From  the  time  an  Air  Force  pilot  completed  IQT  until  they  are  certified  as  “Combat 
Mission  Ready”  they  are  basically  required  to  completed  a  variety  of  formal  and  information 
training  and  operational  requirements  as  a  part  of  the  Ready  Aircrew  Program  (RAP)  However 
the  specificity  of 

Crane  (1999)  reported  that  F-16  instructor  pilots  participating  in  DMT  research  would  frequently 
fly  as  flight  leader  for  the  morning  mission  but  would  then  assign  a  less  experienced  pilot  to 
serve  as  flight  lead  during  the  afternoon.  The  instructors  explained  that  these  pilots  were 
participating  in  Flight  Lead  Upgrade  (FLUG)  training  at  their  squadrons  and  that  DMT  was  a 
valuable  complement  to  aircraft  training.  As  applied  to  FLUG  training,  results  to  date 
demonstrate  that  experience  in  DMT  enhances  leadership  skills  for  multi-ship,  multi-bandit  air 
combat.  This  is  most  likely  a  function  of  both  DMT  system  capabilities  and  constraints  on 
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pilots’  ability  to  train  in  the  aircraft  that  limit  opportunities  for  gaining  experience  over  a  wide 
range  of  tactical  situations.  Observations  of  pilots  during  DMT-FLUG  exercises  demonstrated 
that  increases  in  mission  performance  resulted  from  both  enhanced  skills  and  from  opportunities 
to  review  missions  and  to  assess  the  results  of  individual  actions. 

One  of  the  objectives  of  DMT-FLUG  research  was  to  identify  the  ways  in  which  DMT 
experience  could  increase  pilot  readiness  for  FLUG  missions.  These  examples  illustrate  two 
training  benefits  of  DMT.  The  first  is  providing  increased  opportunities  to  train  individual  skills 
that  must  take  place  in  a  multi-player  context.  Radar  sorting  and  targeting  requires  multiple 
friendly  and  enemy  aircraft;  opportunities  to  practice  this  skill  in  the  air  are  highly  limited.  DMT 
provides  many  opportunities  to  practice  this  skill  in  a  naturalistic  context  and  within  a  short 
period  of  time.  This  observation  is  in  agreement  with  Bell  (1999)  and  Schneider  (1989)  who 
propose  that  multiple  opportunities  to  practice  a  skill  in  a  variety  of  contexts  will  result  in 
substantial  performance  enhancement.  A  second  training  benefit  from  DMT  is  the  opportunity 
gain  experience  in  making  decisions.  Klein  (2000)  asserts  that  lack  of  experience  is  the  primary 
reason  for  poor  decision  making.  Without  high  levels  of  experience,  decision  makers  often  fail 
to  recognize  the  implications  of  various  situations  and  select  inappropriate  courses  of  action. 
Experience  in  DMT  can  augment  aircraft  experience  by  providing  pilots  with  multiple 
opportunities  to  execute  various  tactics  and  to  review  the  results  of  their  actions. 

In  the  next  iteration  of  the  DMT-FLUG  training  effort,  we  will  be  developing  more 
comprehensive  training  and  measurement  capabilities  that  will  permit  us  to  identify  which 
behaviors  improve  and  how  DMT  helped  in  the  improvement  process.  Our  findings  to  date 
indicate  that  minimum  capabilities  for  effective  DMT  include  the  following: 

•  DMT  systems  must  be  supported  by  programmable  scenario  generation  tools  that  permit  the 
incorporation  of  instructional  principles  and  training  strategies  to  foster  skill  development  and 
retention. 

•  Mission  replay  and  after-action  reviewing  capability  are  necessary  capabilities  for  an 
instructionally  viable  training  and  rehearsal  system. 

•  A  construct-oriented  individual,  team  and  mission  measurement  system  in  place. 

Simple  practice  or  free-play  are  not  efficient  training  strategies.  Programmable  scenario 
generation  tools  including  computer-generated  threats  with  the  capability  for  autonomous  action 
permit  instructors  to  take  advantage  of  findings  from  research  on  instructional  principles  to 
design  training  and  rehearsal  events  that  will  meet  specified  objectives.  In  multi-ship  air  combat, 
training  objectives  for  upgrading  pilots  include  learning  to  recognize  enemy  aircraft  formations 
and  selecting  an  appropriate  tactic,  communicating  the  plan  to  the  rest  of  the  flight,  executing  the 
plan,  and  changing  tactics  as  required.  The  programmable  scenario  generation  tools  in  DMT- 
FLUG  were  used  to  create  enemy  formations  and  tactics  that  varied  across  a  number  of 
dimensions  as  summarized  in  Table  2.  This  range  of  scenarios  combined  with  the  opportunity 
for  teams  to  fly  several  scenarios  in  an  hour  provide  upgrading  pilots  with  opportunities  to  gain 
experience  with  many  more  tactics  and  situations  than  would  be  possible  in  the  aircraft. 
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ACDI  tATlf\er  kel  element  in  the  success  of  DMT  is  the  replay  and  debrief  facility.  In  the 
RL  testbed,  each  cockpit’s  radar,  radar  warning  receiver,  head-up  display,  and  stores 
management  system,  are  recorded  and  played  back  synchronized  with  a  plan  view  display  and  all 
radio  communications.  The  flight  leader  can  pause,  rewind,  or  zoom  the  display  as  required 
Using  the  replay  system,  pilots  can  review  the  information  that  was  available  to  them  inside  the 
ckpit  together  with  the  plan  view  display’s  depiction  of  the  complete  tactical  situation. 
Execution  errors,  poor  communication,  and  unplanned  contingencies  are  quickly  apparent 
Debriefs  were  limited  to  90  minutes  for  scheduling  purposes  which  forced  teams  to  focus  on 

and  excci,,ion  withom  getting  absorbed  in,° anaiysis  °f 

Finally,  it  is  important  to  note  that  the  results  presented  in  this  report  represent  an 
evolution  of  our  training  paradigm  from  one  of  structured  practice  to  one  that  is  more  focused  on 
both  objectives-dnven  instruction  and  construct  oriented  performance  measurement.  The  lessons 
learned  from  this  initial  DMT-FLUG  effort  have  provided  considerable  insight  into  the  problems 
and  needed  capabilities  associated  with  implementing  a  high  fidelity,  adaptive  training  and 
rehearsal  system  for  Air  Force  pilots.  The  following  summaiy  describes  future  directions  in 
research  being  undertaken  at  AFRL/HEA  based  on  the  results  of  DMT-FLUG. 

Training  needs  assessment  and  content  validation 


In  the  next  iteration  of  DMT-FLUG  research,  we  will  be  conducting  systematic 
assessment  of  training  needs  at  other  F-16  bases.  The  goal  of  the  needs  assessment  is  to 
elaborate  the  specific  tasks  and  mission  areas  that  are  problematic  for  pilots  in  upgrade  training. 
Additionally ,  the  needs  assessment  will  provide  the  necessary  data  to  develop  an  instructionally 
principled  curriculum.  By  focusing  on  key  needs,  we  can  design  a  syllabus  that  fosters  the 
development  and  retention  of  skills  needed  for  FLUG  success.  Moreover,  the  needs  assessment 
will  complement  our  current  research  activities  in  the  definition  of  mission  essential 
competencies,  i.e.,  the  knowledge,  skills,  attitudes,  abilities  and  capabilities  required  for  combat 
mission  readiness.  This  effort  will  help  us  develop  a  more  comprehensive,  objectives-oriented 
syllabus  for  DMT-FLUG  and  identify  the  important  performance  parameters  associated  with 
mission  effectiveness.  It  will  also  help  us  in  the  development  of  metrics  to  quantify  training 
impact  in  terms  of  combat  mission  readiness  and  performance. 

Development  of  Performance  Criteria  and  Measurement  Instruments 


We  are  currently  in  the  process  of  developing  and  testing  evaluation  instruments  for 
individual  and  team  effectiveness  based  on  mission  essential  competencies.  By  adopting  a 
competency-based  approach  to  training  development  and  evaluation,  we  are  developing  data 
collection  and  analysis  methods  for  use  across  the  entire  spectrum  of  DMT  activities  (e.g., 
mission  preparation,  briefing,  execution,  and  debriefing).  Measurement  techniques  are  being 

developed  for  analysis  of  verbal  communications,  pilots’  mental  models,  team  processes  and 
mission  outcomes. 


•  Content  analysis  methods.  Content-analysis  instruments  are  being  developed  to  evaluate 
verbal  communications  in  order  to  better  understand  the  quality  of  briefings,  within-mission 
communication,  and  debriefings.  In  addition,  measures  of  attitudes  toward  training, 
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motivation  and  perceived  application  and  utility  of  the  training  have  been  developed  for  use  in 
other  training  research  and  will  be  extended  to  FLUG. 

•  Mental  modeling  assessments.  We  are  also  conducting  a  series  of  studies  on  structured 
assessments  of  learning  such  as  situational  judgement  assessments,  mental  modeling, 
Pathfinder,  concept  mapping,  and  multidimensional  scaling.  These  are  being  applied  to  assess 
learning  and  understanding  at  both  individual  and  team  levels. 

•  Process  and  outcome-oriented  measurement.  Using  mission  essential  competency 
definitions,  it  is  possible  to  identify  process-oriented  behaviors  that  are  associated  with 
effective  mission  performance.  We  will  then  be  able  to  develop  process-oriented  measures 
that  will  help  us  quantify  the  effects  of  DMT  experience  on  individual  and  team  knowledge 
acquisition,  crew  coordination/flight  integrity  (including  leadership  and  assertiveness), 
situational  awareness,  communication,  decision  making/risk  management,  task  management, 
and  mission  planning/debrief.  As  a  part  of  these  activities,  we  will  develop,  test,  and  validate 
protocols  to  capture  data  related  to  the  identified  processes.  The  key  process  behaviors  and 
metrics  will  be  used  to  further  refine  training  objectives  and  to  identify  opportunities  to 
improve  training  content  and  training  policy. 

Finally,  we  are  evaluating  the  reliability  and  validity  of  mission  outcome  measures  such 
as  “Turkey  Shoot”  and  “Top  Gun”  scoring.  Based  on  this  evaluation,  we  will  be  refining, 
automating,  and  applying  measures  for  simulator  and  flying  performance.  It  is  critical  that  any 
developed  measurement  system  provide  reliable  and  valid  data  to  demonstrate  that  training  has 
had  an  impact  on  actual  aircrew  competencies,  attitudes  toward  training,  motivation  to  train, 
expectations  regarding  training  and  transfer,  knowledge  mastery  and  subsequent  job 
performance.  Furthermore,  the  data  and  instrumentation  will  serve  both  a  formative  assessment 
function  by  providing  feedback  on  problem  areas  in  the  training,  and  a  summative  function  by 
permitting  cost  and  mission  impact  assessments. 

Summary 

DMT  research  has  demonstrated  both  the  utility  and  effectiveness  of  DMT  to 
complement  aircraft  training.  DMT  supports  operational  training  needs  training  by  providing 
opportunities  for  pilots  to  plan,  brief,  fly,  and  debrief  many  more  air-to-air  missions  than 
currently  possible  in  aircraft.  In  addition,  DMT  provides  instructors  the  capability  to  choose 
scenarios  that  exercise  selected  skills.  New  research  will  focus  on  developing  a  comprehensive 
training  syllabus  that  will  incorporate  needs  analysis,  specification  of  essential  competencies, 
scenario  development  and  selection  based  on  principles  of  instruction,  and  performance 
measurement  systems  that  will  assess  both  mission  outcome  and  team  processes. 
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Background 


Aircraft  and  system  acquisitions  are  expensive  investments,  both  in  terms  of  initial 
acquisition  and  lifecycle  costs.  A  primary  focus  of  most  new  system  acquisition  programs 
is  “affordability.”  Specifically,  this  entails  reducing  the  costs  of  development,  production, 
and  ownership  through  definition,  demonstration,  and  transition  of  innovative  concepts 
and  advanced  technologies  that  will  significantly  reduce  system  costs  while  meeting  the 
operational  requirements  of  the  customers  (i.e.,  the  services).  One  strategy  being  used  to 
reduce  costs  is  to  maximize  commonality  among  air  vehicle  variants  and  systems,  while 
providing  unique  features  that  maximize  capabilities  for  diverse  missions.  Another 
strategy  for  aircraft  acquisition  includes  eliminating  the  need  for  a  two-seat  aircraft 
version  for  training.  This  would  provide  cost  savings  in  terms  of  design,  production, 
maintenance,  and  manpower.  However,  in  order  to  ensure  that  there  would  be  no 
reduction  in  training  effectiveness  or  safety,  a  methodology  needed  to  be  developed  to 
examine  the  purpose  of  current  two-seat  training  and  to  explore  training  and  aircraft 
design  options  that  might  help  mitigate  the  need  for  this  expensive  training  variant.  The 
purpose  of  this  paper  is  to  describe  a  methodology  for  collecting  field  data  to  support  the 
acquisition  and  design  process. 


Objectives 

The  research  team  was  tasked  to  conduct  a  study  analyzing  elements  of  new  aircraft  pilot 
training  that  would  potentially  be  impacted  by  a  decision  not  to  build  a  two-seat  training 
variant.  The  goal  of  this  study  was  to  assist  acquisition  decision  makers  in  the 
Engineering,  Manufacturing,  and  Development  phase  by  identifying  the  critical  pilot 
training  issues  of  safety  and  effectiveness  for  tasks  currently  trained  in  two-seat  aircraft 
variants.  This  study’s  findings  and  recommendations  will  help  the  program  offices  ensure 
that  the  critical  issues  are  adequately  addressed  and  resolved  by  the  weapon  system 
contractors  through  aircraft  design  or  training  system  innovation. 

This  study  gathered  key  information  about  appropriate  training  strategies  from  instructor 
pilots  (IPs)  of  existing  strike  aircraft  of  several  services.  This  information  will  be  used  to 
generate  recommendations  to  ensure  that  new  aircraft  and  training  system  designs  will 
mitigate  any  risks.  Several  training  strategies  and  aircraft  configurations  are  plausible  and 
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combinations  of  strategies  may  provide  the  best  options  for  the  various  aviation 
communities.  To  enhance  the  validity  of  decisions  made  based  on  information  derived 
from  this  study,  this  project  used  subject  matter  expert  (SME)  input  from  current  strike 
fighter  aircraft  IPs  from  various  services  and  countries  participating  in  new  aircraft 
acquisition  programs.  There  are  two  primary  objectives  of  this  survey: 


Objective  1  -  Determine  why  certain  tasks  for  current  aircraft  are  trained  in  2-seat 
models. 

Objective  2  -  Determine  what  training  methods  could  be  used  to  safely  and 
effectively  train  tasks  currently  trained  in  a  2-seat  aircraft  if  there  are  no  2-seat 
aircraft  available. 


Methods 


Workshop 

The  project  team  for  this  study  consisted  of  two  occupational  survey  analysts,  an 
industrial/organizational  psychology  professor  from  Texas  A&M,  and  two  research 
psychologists  who  specialize  in  U.S.  Air  Force  and  Navy  aviation  training.  The  project 
team  conducted  a  two-day  workshop  with  a  diverse  group  of  IP  subject  matter  experts 
to  construct  the  appropriate  survey  items  and  response  options  to  meet  the  defined 
objectives.  The  IPs  represented  legacy  strike  fighter  aircraft  communities.  Specifically, 
the  instructor  pilots  consisted  of: 

a  UK  Royal  Navy  F/A2  former  IP)  who  works  at  a  new  aircraft  program  office  as 
the  lead  for  UK  operational  requirements; 

two  USAF  F-16  pilots  who  were  also  exchange  pilots  with  the  RAF  flying  GR7s, 
one  of  whom  was  an  IP  for  F-16  initial  qualification  training  and  the  other  was  an 
IP  for  undergraduate  pilot  training; 

one  USN  F/A-18  IP;  and 

one  USMC  AV-8B  IP  who  accomplished  the  task  and  syllabus  matching  off-line 
from  the  workshop. 


The  project  team  began  the  workshop  by  giving  an  overview  of  a  new  aircraft  acquisition 
program  and  its  various  roles/missions,  performance  characteristics,  and  planned  system 
capabilities.  The  project  team  then  defined  for  the  SMEs  the  objectives  of  the  study  and 
suggested  types  of  survey  items  that  might  be  appropriate  for  answering  those  objectives. 
The  IPs  reviewed  each  of  their  respective  services'  aircraft  training  syllabi,  course  training 
standards,  and  master  training  task  lists  to  identify  the  tasks  and  skills  currently  trained  in 
two-seat  versions  of  their  aircraft.  The  IPs  then  generated  a  single  list,  using  common 
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terminology,  of  all  the  major  tasks  that  are  trained  in  the  various  services’  two-seat 
aircraft. 

Having  a  common  list  will  enable  an  analysis  of  the  data  across  the  different  services  for 
tasks  that  are  common  to  several  or  all  of  the  services.  The  master  list  contained  44  two- 
seat  tasks,  of  which,  the  USAF  F-16  train  33,  the  USN  and  TJSMC  F/A- 18  train  26,  the 
USMC  AV-8B  train  28,  and  the  UK  RN  and  RAF  train  43.  In  addition,  each  IP  identified 
the  appropriate  legacy  syllabus  event  in  which  those  tasks  are  trained,  and  what  type  of 
instructor  pilot  is  responsible  for  teaching  that  task. 

To  meet  the  first  objective  of  the  study,  determining  why  certain  tasks  are  currently 
trained  in  two-seat  models,  the  project  team  and  SME's  constructed  a  set  of  survey  items 
that  asked  respondents  to  assess  the  importance  of  each  of  the  four  roles  for  an  IP  who  is 
in  the  back-seat  while  training  each  of  the  identified  two-seat  tasks.  The  four  major 
functions  of  the  IP  were  determined  to  be  (a)  Ensure  Safety  of  Flight,  (b)  Demonstrate 
Execution  of  Task,  (c)  Provide  Immediate  Feedback/Instruction,  and  (d)  Assess 
Performance.  Since  an  IP  often  performs  more  than  one  of  these  functions  concurrently 
from  the  back  seat,  we  asked  the  survey  respondents  to  rate  the  importance  of  each 
function  for  each  specified  task  on  a  scale  of  0  to  4  (0  being  “Not  at  all  important”  and  4 
being  “Absolutely  Important”). 

Since  new  aircraft  might  only  have  single-seat  versions  of  the  aircraft,  even  for  training, 
this  study  collected  data  from  survey  participants  on  potential  aircraft  capabilities  or 
training  system  technologies  that  might  be  required  to  train  these  types  of  tasks  with 
minimal  levels  of  risk  and  high  levels  of  training  effectiveness.  With  the  combined 
knowledge  of  the  project  team  and  SMEs,  a  list  was  generated  with  supporting  definitions 
of  aircraft  capabilities  that  might  enable,  or  obviate  the  need  for,  an  IP  to  perform  the  four 
functions  identified  above  in  the  absence  of  a  two-seat  training  aircraft.  These  aircraft 
capabilities  included  such  items  as  a  chase  aircraft,  ground-collision  avoidance  system, 
and  advanced  handling  characteristics  (See  Table  1). 

In  addition  to  the  list  of  aircraft  capabilities,  the  project  team  and  SMEs  developed  a  list  of 
simulation  technologies  with  supporting  definitions,  for  both  existing  and  emerging 
concepts,  that  could  be  used  to  facilitate  or  enhance  training,  or  for  evaluation  of 
competency  before  a  student  must  perform  those  tasks  in  the  aircraft  (see  Table  1). 
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Tgbte  L  List  of  aircraft  and  simulator  capability  response  options  used  on  the  survey 


Aircraft  Capabilities 

Simulator  Capabilities 

No  enhanced  aircraft  capabilities  or  instructor  in 
another  aircraft  required 

Not  Required  For  Training 

Instructor  in  Another  Aircraft 

Basic  Simulator 

In-Flight  Data  Link 

High  Physical  and  Functional  Fidelity  Cockpit 

Ground  Data  Link 

Photo-Realistic  Visual  Database 

Ground  Collision  Avoidance  System  (GCAS) 

360-Degree  FOV  Visual  Displays  with  20/20 
Resolution 

Advanced  Handling  Characteristics 

Realistic  Weather  Effects 

Mission  Recorder 

Networked  Simulators 

Automation 

Accurate  Sensor  Models 

Other 

Motion-Based 

r " "  "  ■■  ■■  "t 

I?6"*  ■.  ■  -  ;  .  V  , 

Force  Cueing 

Other 

Survey  Construction  and  Layout 

Because  of  the  requirement  to  survey  populations  at  disparate  sites,  as  well  the  uniqueness 
of  the  survey  based  on  the  different  tasks  from  the  master  list  that  are  actually  relevant  to 
the  different  services,  the  project  team  used  an  adaptive,  internet-based  survey 
construction  and  delivery  tool  called  GenSurv  (Bennett,  1999;  Mitchell  &  Weissmuller 
1999).  This  survey  tool  allowed  the  survey  to  branch  based  on  the  service,  aircraft  type, 
and  instructor  pilot  qualifications. 

The  survey  began  by  explaining  the  purpose  of  the  study  and  providing  some  background 
information  on  the  new  aircraft,  to  include  its  expected  missions  and  capabilities.  The 
survey  then  had  a  section  to  gather  demographic  information  from  each  respondent  which 
was  branched  and  tailored  to  each  service.  The  next  section  was  constructed  to  collect  data 
to  address  the  first  objective,  assessing  the  importance  of  the  IP  functions  for  each  task 
currently  trained  in  two-seat  training  aircraft.  This  section  began  by  defining  the 
functions  and  then  asked  respondents  to  assess  the  importance  for  each  of  the  tasks 
relevant  to  their  legacy  aircraft  type  and  their  instructor  qualifications. 

The  next  section  addressed  the  second  objective  of  determining  aircraft  and  simulation 
capabilities  that  would  be  required  to  safely  and  effectively  train  those  same  tasks  with 
only  a  single-seat  aircraft  available.  It  began  by  having  the  respondent  read  through  the 
response  options  and  their  detailed  definitions.  Participants  were  then  asked  to  identify 
any  and  all  of  the  aircraft  and  simulation  capabilities  they  thought  would  be  required  for 
each  relevant  task  (see  Table  1). 
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The  final  section  included  a  general  comment  section  for  respondents  to  provide  any 
additional  inputs  on  these  training  issues  they  felt  might  be  useful  and  gave  an  option  for 
the  respondents  to  provide  personal  contact  information. 

In  addition  to  being  able  to  create  an  adaptive  survey  based  on  the  branching  described 
above,  there  are  other  features  of  GenSurv  that  were  extremely  useful  for  constructing  the 
survey  and  using  the  instrument  to  collect  the  data.  GenSurv  allowed  the  survey 
developers  to  require  an  appropriate  response  before  a  participant  could  proceed  to  the 
next  item.  Another  key  feature  used  throughout  the  survey,  was  allowing  for  or  requiring, 
open-ended  responses  or  comments  which  helped  to  collect  a  richer  set  of  information  for 
the  study.  Because  the  survey  is  Internet-based,  the  instrument  developers  inserted  links  to 
a  new  aircraft  program  office  web  site  for  respondents  seeking  more  information.  There 
were  also  pictures  from  simulation  visual  databases  to  illustrate  some  of  the  advanced 
simulation  capabilities  that  new  aircraft  training  systems  could  employ. 

Alpha  and  Beta  Tests 

After  the  survey  was  developed  with  GenSurv  and  hosted  on  a  server,  an  alpha  test  of  the 
survey  was  conducted  with  the  project  team  and  SMEs  who  helped  develop  the  survey. 
Based  on  feedback  from  project  team  members  and  SMEs,  changes  were  made  to  clarify 
the  objectives  and  to  enhance  the  presentation  and  usability  of  the  survey.  A  limited  beta 
test  was  also  conducted  with  IPs  who  were  not  part  of  the  survey  development  to  ensure 
usability  and  clarity  to  pilots  of  the  targeted  population,  and  to  ensure  complete  and 
accurate  coverage  of  two-seat  tasks  for  each  aircraft  community.  The  feedback  from  the 
beta  test  showed  that  the  survey  was  ready  for  implementation  across  the  targeted 
population. 


Field  Data  collection 

The  project  team’s  goal  for  the  survey  was  to  generate  a  sample  of  at  least  thirty  current 
or  recent  instructor  pilots  from  each  legacy  aircraft  community,  resulting  in  a  final  sample 
of  120  participants.  Data  collection  began  on  September  7  and  the  web  site  was  closed 
on  October  6,  due  to  program  schedule  constraints.  At  the  conclusion  of  the  data 
collection,  48  completed  survey  responses  had  been  received  from  the  following  aircraft 
communities:  8  from  United  Kingdom  GR7  and  F/A2  IPs,  10  from  AV-8B  IPs,  18  from 
F-16  IPs,  and  12  from  F/A-18  IPs. 


Conclusion 

Analysis  of  the  data  has  just  begun.  Currently,  the  background  data  and  raw  database  are 
being  evaluated  for  completeness.  Results  are  expected  to  highlight  areas  that  may 
require  risk  mitigation,  either  through  innovative  aircraft  design  or  implementation  of 
innovative  training  technologies  and  strategies.  Results  may  indicate  a  need  to  extend  the 
data  collection  effort  to  increase  the  sample  size  or  highlight  areas  that  require  a  more 
detailed  study  and  analysis  to  fully  understand  the  critical  safety  and  training  effectiveness 
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issues.  The  data  need  to  be  summarized  in  such  a  way  as  to  represent  the  professional 
opinions  of  this  sample  of  pilots  as  to  how  training  should  be  conducted  with  new 
systems.  This  will  assist  new  aircraft  acquisition  decision  makers  in  guiding  the 

development  of  the  aircraft  and  its  pilot  training  systems  to  mitigate  risks  and  ensure 
effective  training. 
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